Study of 3-Day Partial Breast Radiation Therapy in Women With Breast Cancer

Inclusion Criteria: * Female * Age \>/= 45 years * Unicentric pathological stage I (pT1 or T2 pN0\_M0) invasive ductal breast cancer or DCIS measuring \<3 cm in longest diameter on pathology and/or mammogram that is histologically confirmed. If T2, the tumor must be less than 3cm in longest diameter. Note: Women ≥ 70 years or older with T1 invasive ductal carcinoma who are estrogen- receptor positive (ER+) with clinically negative axillary nodes, and do not undergo surgical lymph node mapping or dissection (i.e., if tumor deposit is 0.2 mm or less, regardless of whether the deposit is detected by immunohistochemistry or hematoxylin and eosin staining) will also be eligible. Patients age 50 or older who are clinically node-negative by physical exam and ultrasound and have a primary tumor that is \<=2 cm and hormone-receptor positive, are also eligible. * Histologically negative tumor margin or no tumor in a re-excision specimen on final shaved specimen. * ECOG Performance Status of 0 or 1. * Negative serum pregnancy test within 14 days prior to study treatment if a woman has child-bearing potential. Subjects of child bearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year * Written informed consent obtained from subject and ability for subject or comply with the requirements of the study * Female subjects of childbearing potential should be willing to use 2 methods or birth control or be surgically sterile or abstain from heterosexual activity for the duration of study participation. Should a woman become pregnant while participating on study, she should inform the treating physician immediately. Inclusion Criteria for intermediate risk substudy: * Post-NAC cohort patients: clinical T1-T2 (less than or equal to 5cm in longest diameter on available imaging) and clinical N0-N1 (on exam and imaging) with pathological ypT0, ypTis, or ypT1 (\<=2cm) and ypN0 (any regimen of neoadjuvant chemotherapy agents is allowed) OR * Oncotype RS score of 26 or higher OR * PAM50 ROR scored as "HIGH" OR * Presence or LVI (focal, limited or "not otherwise specified") in the lumpectomy specimen OR * Age 40-49 years (ALL OF THE FOLLOWING MUST BE TRUE : 1) no history of prior benign breast biopsies, 2) no concomitant or prior atypia in either breast, 3) no concomitant or prior LCIS in either breast, 4) no family history of breast cancer in first degree relatives) OR * Invasive lobular carcinoma OR * Surgical margins less than 1 mm to invasive or in-situ disease. Exclusion Criteria: * Patients with distant metastasis * Patients who are pregnant or breastfeeding * Patients with diffuse (\>1 quadrant or \>5cm) suspicious microcalcifications or patients with known multicentric OR multifocal disease. (microscopic multifocal disease that may be unifocal and/or appear multifocal due to sectioning is allowed after review with PI). * Prior radiation therapy to the ipsilateral or contralateral breast or thorax. * Histological evidence of extensive lymphovascular invasion (LVI) * Histological evidence of extensive intraductal component (EIC), defined as the presence of intraductal carcinoma both within the primary infiltrating ductal tumor (comprising at least 25% of the tumo area) and intraductal carcinoma present clearly beyond the edges of the invasive tumor. * Patients are not required to undergo BRCA1 and BRCA2 or other genetic mutation tests in order to enroll on the study. Note: in the event a patient is tested and is found to be a mutation carrier, she would be excluded from the study. It would be an extremely rare/unlikely scenario for patients to be discovered BRCA positive after the completion of PBI, as all patients with risk factors for BRCA mutations (positive family history, Ashkenazi Jewish descent, ER-/PR-/her2-neu-negative receptor status) are usually tested prior to radiation. Should such a situation exist, these patients will be given the option of remaining on the breast conservation paradigm or opting for mastectomy (as is done in this rare scenario is standard of care practice). The patient will be replaced on the trial. * History of cosmetic or reconstructive breast surgery. * Medical condition such as uncontrolled infection (including HIV), uncontrolled diabetes mellitus, or connective tissue diseases (lupus, systemic sclerosis, or other collagen vascular diseases) that, in the opinion of the treating physician, would make this protocol unreasonably hazardous for the patient. * Patients with a "currently active" second malignancy other than non-melanoma skin cancers. Patients are not considered to have "currently active" malignancies if they have completed therapy and are considered by their physicians to be at \<5% risk of relapse within 3 years. * Patients who are already enrolled in or planning to enroll in other adjuvant systemic therapy protocols for both non-invasive or invasive breast cancer * Expecting to conceive within the projected duration of the trials, starting with screening visit through 180 days after the last dose of trial treatment * Concomitant anti-neoplastic treatment is not allowed during protocol treatment and should be completed at least 2 weeks prior to commencement of protocol treatment, with resolution of associated acute toxicities. Bisphosponates are permitted without restriction even during protocol treatment. Note: This does not apply to hormonal therapies such as tamoxifen or AIs, which are permitted. This does not apply to anti-Her2 therapies such as trastuzumab or TDM-1, which are also permitted. * Ongoing therapy with other investigational agents. Patients may not be receiving any other investigational agents.