Personalized Vaccine Immunotherapy in Combination With Checkpoint Inhibitor for Treatment of Triple Negative Breast Cancer

Inclusion Criteria: * Written informed consent and Health Insurance Portability and Accountability Act (HIPAA) authorization for release of personal health information * Must be age \>= 18 years * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 within 14 days prior to tissue consent * Absolute neutrophil count \> 1500/mcL (obtained within 14 days prior to vaccine administration) * Absolute lymphocyte count \>= 600 cells/µl (obtained within 14 days prior to vaccine administration) * Platelets \> 100,000 mm (obtained within 14 days prior to vaccine administration) * Hemoglobin \> 9.0 g/dL (obtained within 14 days prior to vaccine administration) (NOTE: The use of transfusion or other intervention to achieve hemoglobin \[Hgb\] \> 9.0g/dl is acceptable) * Serum creatinine =\< 1.5 x upper limit of normal (ULN) or calculated creatinine clearance \>= 60 mL/min using Cockcroft-Gault equation for patients with creatinine levels \> 1.5 x institutional ULN (obtained within 14 days prior to vaccine administration) * Total bilirubin =\< 1.5 x ULN OR direct bilirubin =\< 1 x ULN (obtained within 14 days prior to vaccine administration) * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x ULN unless liver metastases are present, in which case they must be =\< 5 x ULN (obtained within 14 days prior to vaccine administration) * Bilirubin =\< 1.5 X ULN (except in participants with documented Gilbert's disease, who must have a total bilirubin =\< 3.0 mg/dL) (obtained within 14 days prior to vaccine administration) * International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 x ULN unless patient is receiving anticoagulant therapy as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants (obtained within 14 days prior to vaccine administration) * Activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless patient is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants (obtained within 14 days prior to vaccine administration) * TNBC as defined by estrogen receptor (ER)/progesterone receptor (PR) =\< 10% if Allred =\< 3; Her2/neu negative as defined by scores of 0 or 1+ by immunohistochemistry (IHC) or 2+ by IHC associated with a fluorescence in situ hybridization (FISH) ratio of \< 2.0 or \< 6 Her2 copies per cell * Patients with metastatic or inoperable locally advanced disease: Metastatic or inoperable locally advanced disease is defined as either histologically confirmed metastatic breast cancer by biopsy; or locally advanced breast cancer that, in the opinion of the treating physician, is not amenable to curative intent surgical resection; or, radiological or clinical evidence suggestive and supportive of metastatic disease * Documented metastatic biopsy is not required provided the patient has a prior diagnosis of TNBC that otherwise meets the eligibility criteria * Eligible patients must have =\< 3 lines of chemotherapy in the metastatic/advanced disease setting. For patients who have relapsed within 6 months of systemic therapy given within curative intent, that therapy will count as a line of metastatic therapy. Last cycle of cytotoxic therapy must be \>= 21 days prior to cycle (C)1 day (D)1 of vaccine. Last cycle of checkpoint inhibitor therapy be \>= 28 days prior to C1D1 of vaccine. Last dose of radiotherapy must be \>= 14 days prior to C1D1 of vaccine * Prior checkpoint inhibitor is permitted. Patients who are known to have PD-L1 positive with combined positive score (CPS) \>= 10 will be required to have had pembrolizumab therapy prior to enrollment * Patients who are metastatic or inoperable, locally advanced will only be eligible for the Phase 1b combination cohort. They will not be eligible for the Phase 1a dose escalation cohort. Patients who have received prior anti-CTLA-4 antibody will preferentially be enrolled to cohort B (combination of TMV vaccine with pembrolizumab). Patients who have received prior PD-L1 or PD-1 antibody therapy will preferentially be enrolled to cohort C (combination of TMV vaccine with ipilimumab). Patients with metastatic disease who have received no prior immune checkpoint inhibitor therapy will be assigned to a treatment arm based on available slots and discretion of treating physician * Patients with early stage TNBC: Early stage TNBC is defined as clinical or pathologic Stage I-III TNBC * After resection of disease in the breast and axilla, early stage patients are eligible for either the Phase 1a dose escalation of TMV vaccine monotherapy Cohort A or the Phase 1b combination arm of the vaccine with immune checkpoint inhibitor (ICI) * Patients will be required to have completed adjuvant radiotherapy (if indicated) \>= 14 days prior to initiation of vaccine on trial * Patients who have residual disease after completion neoadjuvant therapy that proceed with adjuvant capecitabine can enroll \>= 28 days after completion of final dose of capecitabine. Patients electing to enroll onto the Phase 1a Cohort A monotherapy arm can enroll prior to initiation of capecitabine, however must not initiate capecitabine prior to the Week 12 blood draw (measurement of immune biomarkers) on study * Patients who have a germline BRCA 1/2 mutation that meet the Food and Drug Administration (FDA) indication for use of adjuvant Olaparib can enroll \>= 28 days after completion of final dose of olaparib. Patients electing to enroll onto the Phase 1a Cohort A monotherapy arm can enroll prior to initiation of olaparib, however must not initiate olaparib prior to the Week 12 blood draw (measurement of immune biomarkers) on study * Patients who undergo upfront surgery: Patients may initiate injection of vaccine \>= 28 days after completion of final cycle of adjuvant chemotherapy * Patients who have early stage breast cancer that have residual disease after completing neoadjuvant chemotherap