A Study of Radiation Therapy With Pembrolizumab and Olaparib or Other Radiosensitizers in Women Who Have Triple-Negative or Hormone-Receptor Positive/Her2 Negative Breast Cancer

Inclusion Criteria: * Female participants who are at least 18 years of age on the day of signing informed consent with histologically confirmed diagnosis of triple negative breast cancer or ER+/Her2 will be enrolled in this study. * Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1. Evaluation of ECOG is to be performed within 30 days prior to the date of allocation/randomization * Histologically or cytologically-confirmed TNBC (defined as ER \<5%, PR \<5%, HER- 2-neu 0-1+ by IHC or FISH-negative or per MD discretion). * Histologically of cytologically-confirmed ER+ breast cancer, defined as ER 1-100% and HER-2/neu 0-1+ by IHC or FISH-negative * Metastatic or recurrent TNBC. * Metastatic or recurrent ER+ breast cancer * For mTNBC patients, prior receipt of ICI with progression and/or PDL1-negative. PDL-1 status is not used to determine eligibility among mER+BC patients * Note: PDL1-status may be determined on tissues from either primary or mTNBC. Determination of PD-L1 status by any prototype assays is acceptable. PD-L1 status is required or archival tissue must be readily available for testing during screening if status was not previously determined. * The participant (or legally acceptable representative if applicable) provides written informed consent for the trial. * Have measurable disease based on RECIST 1.1. There should be at least one radiographically-confirmed non-bone metastatic lesion that will not undergo RT and is measurable based on RECIST and suitable for repeated measurements. * Lesions situated in a previously irradiated area are considered measurable if progression has been demonstrated in such lesions. * No more than 3 prior lines of systemic therapy (including conventional cytotoxics, targeted therapies, biologics, or other invesigational systemic treatments) for inoperable/recurrent or metastatic disease in the TNBC cohort. A line of treatment in this instance refers to any systemic therapy directed at metastatic TNBC and which was discontinued due to disease progression. Treatment discontinued due to toxicity will not be counted. Systemic therapy previously delivered for hormone-receptor positive or Her2+ breast cancer (that has now switched to TNBC subtype) will not count as a prior line of treatment. Any number of prior lines of treatment for mER+BC are allowed. * At least one tumor site for which palliative RT is considered clinically appropriate. The site under consideration can be a metastatic site, or uncontrolled primary/locally recurrent disease in the breast/chest wall or in the nodes. Prior radiotherapy to the target site is allowed. Investigators should remain within departmental radiotherapy for normal tissue; exceptions should be discussed with the PI. * Have provided archival tumor tissue sample or newly obtained core or excisional biopsy of a tumor lesion. Formalin- fixed, paraffin embedded (FFPE) tissue blocks are preferred to slides. Newly obtained biopsies are preferred to archived tissue. Newly-obtained is defined as a specimen obtained up to 30 days prior to initiation of treatment on Day 1 of pembro. (Note: PI can waive this requirement at his discretion if specimen collection is deemed unfeasible). * A female participant is eligible to participate if she is not pregnant (see Appendix C), not breastfeeding, and at least one of the following conditions applies: 1. Not a woman of childbearing potential (WOCBP) as defined in Appendix C OR 2. A WOCBP who agrees to follow the contraceptive guidance in Appendix C during the treatment period and for at least one month after the last dose of study treatment. * Have adequate organ function as defined in the following table (Table 2). Specimens must be collected within 30 days prior to the start of study treatment. Table 2 Adequate Organ Function Laboratory Values System: Laboratory Value Hematological Absolute neutrophil count (ANC): ≥ 1500/µL Platelets: ≥ 100 000/µL Hemoglobin: ≥ 9.0 g/dL with no blood transfusion in the past 28 days Renal Creatinine: 1.5 x ULN OR Measured or calculatedb creatinine clearance (GFR can also be used in place of creatinine or CrCl): ≥ 51 mL/min Hepatic Total bilirubin: ≤ 1.5 x ULN OR direct bilirubin ≤ ULN for participants with total bilirubin levels \>1.5 x ULN AST (SGOT) and ALT (SGPT): ≤ 2.5 x ULN (≤ 5 x ULN for participants with liver metastases) Coagulation International normalized ratio (INR) OR prothrombin time (PT); Activated partial thromboplastin time (aPTT): ≤ 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is within therapeutic range of intended use of anticoagulants ALT (SGPT)=alanine aminotransferase (serum glutamic pyruvic transaminase); AST (SGOT)=aspartate aminotransferase (serum glutamic oxaloacetic transaminase); GFR=glomerular filtration rate; ULN=upper limit of normal. a - Criteria must be met without erythropoietin dependency and without packed red blood cell (pRBC) transfusion within last 2 weeks. b - Creatinine clearance (CrCl) should be calculated per institutional standard. Note: This table includes eligibility-defining laboratory value requirements for treatment; laboratory value requirements should be adapted according to local regulations and guidelines for the administration of specific chemotherapies. * Participant must agree not to breastfeed during the study or for 180 days after the last dose of study treatment. * Participant receiving corticosteroids may continue as long as their dose is stable for at least 4 weeks prior to initiating protocol therapy. * Ability to swallow (whole) and retain oral medications. Exclusion Criteria: * Receipt of \> 3 lines of systemic therapy in mTNBC patients * A WOCBP who has a positive urine pregnancy test within 72 hours or a serum pregnancy test within 14 days prior to treatment (see Appendix C). If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. * A minim