Radioembolization With Tremelimumab and Durvalumab for Locally Advanced Unresectable or Oligo-Metastatic Intrahepatic Cholangiocarcinoma

Inclusion Criteria: * Age \>= 18 years with body weight \> 30 kg * Histologically or cytologically confirmed, locally advanced intrahepatic cholangiocarcinoma that is not amenable to resection, transplantation, or thermal ablation. Oligometastatic intrahepatic cholangiocarcinoma is also eligible. Specifically, such patients must have EITHER =\< 3 malignant extrahepatic lymph nodes (short axis diameter \>= 3cm) OR metastatic lesions in one organ other than liver (if only single lesion is present diameter MUST be \< 3cm, if up to 3 lesions in one organ each lesion MUST be =\< 1cm) * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Hemoglobin \>= 9.0 g/dL (=\< 14 days prior to registration) * Absolute neutrophil count (ANC) \>= 1000/mm\^3 (=\< 14 days prior to registration) * Platelet count \>= 75,000/mm\^3 (=\< 14 days prior to registration) * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (patients with known Gilbert disease who have serum bilirubin level 3 x ULN may be enrolled) (=\< 14 days prior to registration) * Alanine aminotransferase (ALT) and aspartate transaminase (AST) =\< 5 x ULN (=\< 14 days prior to registration) * Calculated creatinine clearance \>= 40 ml/min using the Cockcroft- Gault formula or measured creatinine clearance \> 40 ml/min (=\< 14 days prior to registration) * International normalized ratio (INR) =\< 1.6. Note: INR prolongation due * Anticoagulation (INR \>= 2.0 but =\< 3.0)) for prophylaxis in patients without liver cirrhosis could be exception * Adequate hepatic function Child Pugh A and albumin-bilirubin (ALBI) 1 or 2 * Patients with concurrent hepatitis B (HBV) or hepatitis C virus (HCV) infection should meet the following criteria: * Patient with HBV or should be monitored for viral levels during study participation * Patient with detectable hepatitis B surface antigen (HBsAg) or detectable HBV DNA should have HBV DNA \< 100 IU/ml and should be managed per local guidelines * Controlled hepatitis B subjects will be allowed if they have started treatment prior to or by the time point of enrollment into the study and treatment is continued during study participation and for \>= 6 months after end of study treatment * Patients positive for HCV antibody are eligible only if polymerase chain reaction is negative for HCV RNA. * Negative urine pregnancy test done prior =\< 7 days registration, for persons of childbearing potential only * NOTE: If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required Exclusion Criteria: * Concurrent enrollment in another clinical study, unless it is an observational clinical study or during the follow up period of an interventional study * Surgery =\< 28 days prior to registration * Chemotherapy =\< 4 weeks prior to registration * History of \> 1 prior systemic therapy for cholangiocarcinoma not including that in the adjuvant setting. Patients who progressed during or =\< 6 months from completion of adjuvant therapy are excluded * Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the study physician * Patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab or tremelimumab may be included only after consultation with the study physician * History of previous locoregional therapy * Previous use of therapeutic cancer vaccines * Unstable liver function and/ or a change in Child Pugh score during screening * Child Pugh B or greater * ALBI grade \> 2 * Model for End-Stage Liver Disease (MELD) \> 10 * Patient is unable to undergo mapping angiography or mapping angiography demonstrates tumor blood supply that does not lend itself to transarterial therapy * A lung shunt fraction greater than 30 Gy within a single session, or cumulative does greater than 50Gy * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Immunocompromised patients and patients known to be human immunodeficiency virus (HIV) positive and currently receiving antiretroviral therapy * NOTE: Patients known to be HIV positive, but without clinical evidence of an immunocompromised state, are eligible for this trial * Active or uncontrolled autoimmune or inflammatory disorders (including Inflammatory bowel disease, systemic lupus erythematosus, rheumatoid arthritis, granulomatosis with polyangiitis, sarcoidosis, Grave's disease) * History of another primary malignancy except for: * Malignancy treated with curative intent and with no known active disease \>= 5 years prior to registration and of low potential of recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease * Uncontrolled intercurrent illness including, but not limited to: * Ongoing uncontrolled infections including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis (TB) testing in line with local practice), human immunodeficiency virus (HIV), hepatitis B and hepatitis C * Serious chronic gastrointestinal condition associated with diarrhea * Symptomatic congestive heart failure * Unstable angina pectoris, cardiac arrhythmia and uncontrolled hypertension * Chronic pulmonary disease including interstitial lung disease requiring oxygen * Psychiatric illness/social situations limiting complian