Ivermectin in Combination With Balstilimab or Pembrolizumab in Patients With Metastatic Triple Negative Breast Cancer and HR+/HER2- Metastatic Breast Cancer

Inclusion Criteria: * Age: ≥ 18 years * Eastern Cooperative Oncology Group (ECOG) ≤ 1 * Life expectancy \> 3 months * At least one biopsy proven Triple Negative Breast Cancer (TNBC) or HR+/HER2- breast cancer in the metastatic setting. TNBC is defined by ER and PR ≤ 10% and HER2 negative (by IHC or FISH), per ASCO/CAP guidelines. In the case of discordant expression results, eligibility must be discussed and determined at the discretion of the study PI. * Patients must have progressed on 1-2 prior lines of systemic therapy (chemotherapy and/or drug-antibody conjugate) in the metastatic setting * Measurable or evaluable disease by Response Evaluation Criteria in Solid Tumors (RECIST) 1.1. Lesions situated in a previously irradiated area are considered measurable if progression has been demonstrated in such lesions * Fully recovered from the acute toxic effects (except alopecia) to ≤ grade 2 from prior anti-cancer therapy * Patients must have adequate organ function as defined in the following: * Absolute neutrophil count (ANC) ≥ 1,500/mm\^3 * Platelets ≥ 100,000/mm\^3 * Hemoglobin ≥ 9.0 g/dL or ≥ 5.6 mmol/L. Criteria must be met without erythropoietin dependency and without packed red blood cell (pRBC) transfusion within last 2 weeks * Total serum bilirubin ≤ 1.5 x upper limit of normal (ULN) OR direct bilirubin ≤ ULN for participants with total bilirubin levels \> 1.5 x ULN * Aspartate aminotransferase (AST) ≤ 1.5 x ULN or ≤ 3 x ULN with liver metastases * Alanine aminotransferase (ALT) ≤ 1.5 x ULN or ≤ 3 x ULN with liver metastases * Creatinine ≤ 1.5 x ULN OR calculated creatinine clearance ≥ 30 mL/min for participant with creatinine levels \>1.5 x institutional ULN * International normalized ratio (INR) or prothrombin time (PT), activated partial thromboplastin time (aPTT) ≤ 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is within therapeutic range of intended use of anticoagulants * Women of childbearing potential (WOCBP): negative urine or serum pregnancy test \* If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * A male participant must agree to use a contraception during the treatment period and for at least 120 days after the last dose of study treatment and refrain from donating sperm during this period * A female participant is eligible to participate if she is not pregnant, not breastfeeding, and at least one of the following conditions applies: * Not a woman of childbearing potential (WOCBP) OR * Females of child-bearing potential must be willing to use effective contraception during study and for 120 days after the last dose * Written informed consent obtained from subject and ability for subject to comply with the requirements of the study. Exclusion Criteria: * Has a diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of study drug * Prohibited Treatments and/or Therapies: * Chemotherapy, radiation therapy, biological therapy, immunotherapy within 28 days prior to day 1 of protocol therapy * Prior radiotherapy within 2 weeks of start of study intervention. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. A 1-week washout is permitted for palliative radiation (≤2 weeks of radiotherapy) to non-CNS disease * Any live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette-Guérin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (eg, FluMist®) are live attenuated vaccines and are not allowed * Participants on any dose of warfarin. Use of low molecular weight heparin, antithrombin agents, anti-platelet agents or factor Xa inhibitors is allowed * Participants may not be currently participating in or participated in a study of an investigational agent or has used an investigational device within 4 weeks prior to the first dose of study intervention * Issues with tolerating oral medication (e.g., inability to swallow pills, malabsorption issues, ongoing nausea or vomiting during screening) * Women who are or are planning to become pregnant or breastfeed * Known allergy to any of the components within the study agents and/or their excipients * No other prior malignancy is allowed except for the following: adequately treated basal cell or squamous cell skin cancer, in situ cervical cancer, adequately treated stage I or II cancer from which the patient is currently in complete remission, or any other cancer from which the patient has been disease free for at least three years * Participants must not have known active CNS metastases and/or carcinomatous meningitis. Participants with previously treated brain metastases may participate provided they are radiologically stable, i.e., without evidence of progression for at least 4 weeks by repeat imaging (note that the repeat imaging should be performed during study screening), clinically stable and without requirement of steroid treatment for at least 14 days prior to first dose of study intervention * History of (non-infectious) pneumonitis that required steroids or has current pneumonitis * Active infection requiring systemic therapy * Known history of Human Immunodeficiency Virus (HIV) infection * Known history of Hepatitis B (defined as Hepatitis B surface antigen \[HBsAg\] reactive) or known active Hepatitis C virus (defined as HCV RNA \[qualitative\] is detected) infection. Note: no testing for Hepatitis B and Hepatitis C is required unless mandated by local health authority * Known history