Testing the Addition of Stereotactic Radiation Therapy With Immune Therapy for the Treatment of Patients With Unresectable or Metastatic Renal Cell Cancer, SAMURAI Trial

Inclusion Criteria: * Pathologically (histologically or cytologically) proven diagnosis of renal cell carcinoma prior to registration * Node-positive unresectable (TxN1Mx) or metastatic (TxNxM1) based on the following diagnostic workup: * History/physical examination within 45 days prior to registration * CT/magnetic resonance imaging (MRI) of the chest/abdomen/pelvis within 45 days prior to registration * Patients must have IMDC intermediate (1-2 factors) or poor risk disease (\>= 3 factors) * Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial * Patients with measurable disease (node positive or metastatic) as defined by RECIST version 1.1 excluding the primary renal tumor * Patient not recommended for or refused immediate cytoreductive nephrectomy * Candidate for standard of care therapy with either immuno-oncology (IO)-IO or IO-VEGF combination regimen * Primary renal tumor measuring 20 cm or less in anterior to posterior dimension only on axial imaging * Age \>= 18 * Karnofsky performance status \>= 60 within 45 days prior to registration * Hemoglobin \>= 8 g/dL (transfusions are allowed) (within 45 days prior to registration) * Platelet count \>= 50,000/mm\^3 (within 45 days prior to registration) * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (within 45 days prior to registration) * Calculated (Calc.) creatinine clearance \>= 30 mL/min (within 45 days prior to registration) * Creatinine clearance (CrCl) \>= 30 mL/min estimated by Cockcroft-Gault Equation * For African American patients specifically whose renal function is not considered adequate by the formula above, an alternative formula that takes race into account (Chronic Kidney Disease Epidemiology Collaboration CKD-EPI formula) should be used for calculating the related estimated glomerular filtration rate (GFR) with a correction factor for African American race creatinine clearance for trial eligibility, where GFR \>= 30 mL/min/1.73m\^2 will be considered adequate * Total bilirubin =\< 1.5 x upper limit of normal (ULN) (except subjects with Gilbert Syndrome, who can have total bilirubin \< 3.0 mg/dL) (within 45 days prior to registration) * Aspartate aminotransferase and alanine aminotransferase (AST and ALT) =\< 3 x upper limit of normal (ULN) or \< 5 x ULN if hepatic metastases present (within 45 days prior to registration) * Patients with known human immunodeficiency virus (HIV) on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial. Testing is not required for entry into protocol * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. Patients with HCV infection who are currently on treatment are eligible if they have an undetectable HCV viral load * The patient must agree to use a highly effective contraception, including men with vasectomies if they are having sex with a woman of childbearing potential or with a woman who is pregnant, while on study drug and for 6 months following the last dose of study drug. Childbearing potential is defined as any person who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal * The patient or a legally authorized representative must provide study-specific informed consent prior to study entry and, for patients treated in the United States (U.S.), authorization permitting release of personal health information Exclusion Criteria: * Patients with planned treatment of all metastatic disease with definitive therapy including either surgery, ablative (non-palliative) doses of radiation, or intervention of some type (definitive interventional radiology techniques) to ALL metastatic sites rendering the patient without extra-renal measurable disease. Patients NOT planned for definitive treatment of all metastatic sites are eligible. Lesions radiated palliatively are not eligible for response assessment * Patients with untreated or unstable brain metastases or cranial epidural disease * Note: Patients who have been adequately treated with radiotherapy, radiosurgery, or surgery and stable for at least 4 weeks prior to registration as documented by MRI or CT imaging or deemed stable by clinical investigator are eligible. Treated brain metastases are defined as having no ongoing requirement for steroids and no evidence of progression or hemorrhage after treatment for at least 4 weeks prior to registration as documented by MRI or CT imaging or deemed stable by clinical investigator * Prior radiotherapy to the kidney that would result in overlap of radiation therapy fields treatment of the primary tumor * Any systemic therapy for metastatic renal cell carcinoma (RCC) that was initiated \> 90 days before registration, note that prior chemotherapy for a different cancer is allowed (completed \> 3 years prior to registration) * Severe, active comorbidity defined as follows: * Active autoimmune disease requiring ongoing therapy including systemic treatment with corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications daily. Inhaled steroids and adrenal replacement steroid doses \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease * History of severe allergic, anaphylactic or other hypersensitivity reactions to chimeric or humanized antibodies * Active tuberculosis (purified protein derivative \[PPD\] response without active tuberculosis \[TB\] is allowed) * Uncontrolled hypertension (systolic blood pressure \[BP\] \>= 190 mmHg or diastolic BP \> 110 mmHg) * Major surgery requiring hospital ad