TRUDI: TDXD+Durva in HER2+/Low IBC

Inclusion Criteria: * Participants must have a histological or cytological diagnosis of invasive breast cancer. * All histologic subtypes are eligible. * Participants must have a clinical diagnosis of stage III inflammatory breast cancer within the past 6 months * HER2-positive status as determined locally by the current ASCO/CAP guidelines or HER2-low tumor expression (IHC 2+/ISH-, IHC 1+/ISH-, or IHC 1+/ISH untested) (note: ISH may be determined by either fluorescence in situ hybridization \[FISH\] or dual in situ hybridization \[DISH\]) * Any ER and PR expressions are permitted but must be known * Participants must be treatment-naïve * Participants must agree to undergo two research biopsies of the tumor (if safely accessible, as determined by the treating investigator): at baseline (prior to the first treatment) and after the first week of treatment on C1D8. Previously collected archival tissue will also be obtained on all participants. For participants for whom the tumor is not safely accessible, this archival tissue needs to be located and availability confirmed at time of registration. * Pre- and postmenopausal women or male patients ≥ 18 years of age * ECOG performance status 0-1 (Karnofsky \> 60%, see Appendix A). * LVEF ≥ 50% within 28 days prior to enrollment * Participants must have normal organ and marrow function prior to enrollment as defined below: * Absolute neutrophil count ≥2,000/mcL * Platelets ≥100,000/mcL * Hemoglobin ≥ 9.0 g/dl * INR/PT/aPTT ≤1.5 × ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is in therapeutic range of anticoagulant * Total bilirubin ≤1.5 × institutional upper limit of normal (ULN)(or ≤2.0 x ULN in patients with documented Gilbert's Syndrome) * AST(SGOT)/ALT(SGPT) ≤2.5 × institutional ULN * Serum creatinine ≤1.5 × institutional ULN OR creatinine clearance ≥ 30 mL/min/ 1.73m2 for participants with creatinine levels above institutional ULN. * Serum albumin ≥2.5 g/dL * International normalized ratio (INR)/prothrombin time (PT) and either partial thromboplastin or activated partial thromboplastin time (aPTT) ≤1.5 × ULN * Women of childbearing potential (WOCBP) and WOCBP who are partners of male participants must agree to use one highly effective non-hormonal form of contraception or two effective forms of non-hormonal contraception for the duration of study treatment with durvalumab and 7 months after the last dose of study treatment * Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception for the duration of study treatment with durvalumab and 4 months after the last dose of study treatment * Must have a life expectancy of at least 12 weeks * Body weight \>30 kg * The participant must be capable of understanding and complying with the protocol and willing to sign a written informed consent document Exclusion Criteria: * Has received prior systemic anti-cancer therapy for the current diagnosis of inflammatory breast cancer, including chemotherapy, immunotherapy, or targeted therapy * Has received any radiotherapy or surgery for the current diagnosis of inflammatory breast cancer. Tumor biopsies are not considered surgery for the purpose of enrollment. * Prior hypersensitivity to durvalumab or the excipients of durvalumab or trastuzumab deruxtecan or history of severe hypersensitivity reactions to other monoclonal antibodies. * Major surgery within 4 weeks prior to study treatment initiation. Patients must have recovered from any effects of any major surgery. * Uncontrolled intercurrent illness, including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, uncontrolled cardiac arrhythmia, serious chronic gastrointestinal conditions associated with diarrhea, or psychiatric illness/social situations that would limit compliance with study requirements, substantially increase risk of incurring AEs, or compromise the ability of the patient to give written informed consent. * Participant has a medical condition that requires chronic systemic steroid therapy (\> 10 mg of prednisone daily or equivalent) or any other form of immunosuppressive medication (including disease modifying agents) and has required such therapy in the last 2 years. Replacement therapy (e.g. thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic therapy. * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[eg, colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, Wegener syndrome \[granulomatosis with polyangiitis\], Graves' disease, and rheumatoid arthritis, hypophysitis, uveitis, etc). The following are exceptions to this criterion: \-- Patients with vitiligo or alopecia, Patients with hypothyroidism (eg, following Hashimoto syndrome) stable on hormone replacement, Patients with any chronic skin condition that does not require systemic therapy, Patients with celiac disease controlled by diet alone, who may also be included, but only after consultation with the Principal Investigator, Patients without active disease in the last 5 years, who may also be included but only after consultation with the Principal Investigator * History of (non-infectious) ILD/pneumonitis that required steroids, has current ILD/pneumonitis, or where suspected ILD/pneumonitis cannot be ruled out by imaging at screening. * Lung-specific intercurrent clinically significant illnesses including, but not limited to, any underlying pulmonary disorder (ie, pulmonary emboli within 3 months of the study enrollment, severe asthma, severe chronic obstructive pulmonary disease (COPD), restrictive lung disease, pleural effusion etc.), and any autoimmune, connective tissue or inflammatory disorders