a »ÀiÍ^ã@srddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z dd„Z Gdd„dƒZdS)éN)ÚProcessPoolExecutor)Úfile_is_binaryÚsos_get_command_outputÚfile_is_certificatec Csàt |¡Â}tj |d¡}ttddd„ƒ|_| ¡D]d}tj ||j¡}tj  |¡}tj  |¡}tj  ||g¡}||krŒt d|›d|›ƒ‚|  ||¡q4tj ||j  d¡d  d ¡d ¡WdƒS1sÒ0YdS) NZcleanerZfully_trusted_filtercSs|S©N©)ÚmemberÚpathrrúA/usr/lib/python3.9/site-packages/sos/cleaner/archives/__init__.pyÚ#óz!extract_archive..z"Attempted path traversal in tarflez != ú/éÿÿÿÿú.tarr)ÚtarfileÚopenÚosr ÚjoinÚgetattrZextraction_filterZ getmembersÚnameÚabspathÚ commonprefixÚ ExceptionÚextractÚsplit) Ú archive_pathÚtmpdirÚarchiver rZ member_pathZ abs_directoryZ abs_targetÚprefixrrr Úextract_archives$ ÿ   ÿÿrc@sxeZdZdZdZdZdZdZdZdZ iZ dd„Z dd„Z d d „Z d d „Zd d„ZdRdd„Zdd„Zedd„ƒZedd„ƒZedd„ƒZdd„Zdd„Zdd„Zd d!„ZdSd"d#„ZdTd$d%„ZdUd&d'„ZdVd(d)„Zd*d+„Zed,d-„ƒZ d.d/„Z!d0d1„Z"d2d3„Z#d4d5„Z$dWd6d7„Z%d8d9„Z&d:d;„Z'dd?„Z)d@dA„Z*dBdC„Z+dDdE„Z,dFdG„Z-dHdI„Z.dJdK„Z/dLdM„Z0dNdO„Z1dPdQ„Z2dS)XÚSoSObfuscationArchiveaA representation of an extracted archive or an sos archive build directory which is used by SoSCleaner. Each archive that needs to be obfuscated is loaded into an instance of this class. All report-level operations should be contained within this class. rZ undeterminedFcCsž||_|j|_||_|j d¡d d¡d|_|j|_t d¡|_t d¡|_ |  ¡|_ d|_ |  ¡d|_||_||_d |_| d |j›d |j›¡dS) Nr rrrÚsosZsos_uiFÚrzLoaded z as type )rÚfinal_archive_pathrrÚ archive_nameÚui_nameÚloggingZ getLoggerÚsoslogÚui_logÚ_load_skip_listÚ skip_listÚ is_extractedÚ _load_selfÚ archive_rootÚkeep_binary_filesÚtreat_certificatesÚparsersÚlog_infoÚ description)Úselfrrr.r/rrr Ú__init__Cs"   ÿzSoSObfuscationArchive.__init__c CsT|jD]H}z| |¡}WqtyL}z| d|›¡WYd}~qd}~00q|S)NzError obfuscating string data: )r0Zparse_string_for_keysrr1)r3Z string_dataÚparserÚerrrrr Úobfuscate_stringWs  (z&SoSObfuscationArchive.obfuscate_stringcCs’| | d¡d¡}| | d¡d|¡}||krŽ| |¡d}tj ||¡}tj |¡sht ||¡n&| t |¡¡}t  |¡t  ||¡dS)Nr rr) r7rÚreplacerr rÚislinkÚrenameÚreadlinkÚremoveÚsymlink)r3Z short_nameÚfilenameZ_ob_short_nameZ _ob_filenameÚarc_pathZ_ob_pathZ _target_obrrr Úobfuscate_filenameasÿ  z(SoSObfuscationArchive.obfuscate_filenamecCs ||_dSr)r0)r3r0rrr Ú set_parsersxsz!SoSObfuscationArchive.set_parserscCs|jD] }| ¡qdSr)r0Zload_map_entries)r3r5rrr Úload_parser_entries{s z)SoSObfuscationArchive.load_parser_entriesNc Csˆd}| ¡s||fS|dur"|j}|D]X}z| |¡\}}||7}Wq&ty|}z | d|›|j¡WYd}~q&d}~00q&||fS)aÿRun a line through each of the obfuscation parsers, keeping a cumulative total of substitutions done on that particular line. Positional arguments: :param line str: The raw line as read from the file being processed :param parsers: A list of parser objects to obfuscate with. If None, use all. Returns the fully obfuscated line and the number of substitutions made rNzfailed to parse line: )Ústripr0Z parse_linerÚ log_debugr)r3Úliner0Úcountr5Ú_countr6rrr Úobfuscate_lines ,z$SoSObfuscationArchive.obfuscate_linec s¸|D] }| dt ¡›d|›¡z8tjj||jd‰| ˆ¡rHWq|jsf| ˆ¡rf|  ˆ¡Wq|jr‚t |ƒs~| ˆ¡r‚Wqtj  |¡r’Wqt |ƒ}|r|dkr¶|  ˆ¡Wq|j dkrÄWq|j dkrÜ|  ˆ¡Wq|j dkr| |¡}tjj||jd‰‡fdd „|jDƒ}|s:| d ˆp,|›d ¡Wq| d ˆpH|›¡d }tjd|jdÚ}t|dddd€}|D]j}z&| ||¡\}} || 7}| |¡Wn<tyâ} z"| dˆ›d| ›¡WYd} ~ n d} ~ 00q|Wdƒn1sþ0Y| d ¡|r0t |j|¡| |¡Wdƒn1sF0Y| ˆ|¡Wqty¤} z,| dt ¡›d|›d| ›¡WYd} ~ qd} ~ 00q|j|j|jfS)Nz pid=z: obfuscating )ÚstartZcertificatekeyZkeepr<Z obfuscatecs(g|] }t‡fdd„|jDƒƒs|‘qS)c3s|]}| ˆ¡VqdSr)Úmatch)Ú.0Ú_skip©Zrel_namerr Ú ÆszGSoSObfuscationArchive.obfuscate_arc_files...)ÚanyZ skip_patterns)rKZ_prMrr Ú Äs  ÿÿz=SoSObfuscationArchive.obfuscate_arc_files..zSkipping obfuscation of z" due to matching file skip patternz Obfuscating rÚw)ÚmodeÚdirÚrúutf-8r8)ÚencodingÚerrorszUnable to obfuscate ú: z': caught exception on obfuscating file ) rDrÚgetpidr ÚrelpathÚextracted_pathÚshould_skip_filer.Úshould_remove_fileÚ remove_filerr9rr/Úcertificate_to_textr0ÚtempfileZNamedTemporaryFilerrrHÚwriterÚseekÚshutilÚcopyfilerÚupdate_sub_countr@Úfiles_obfuscated_countÚtotal_sub_countÚremoved_file_count) r3Úflistr>Zis_certificateZ_parsersZsubsZtfileÚfnamerEZcntr6rrMr Úobfuscate_arc_files›s–  ÿ ÿþ       ÿ ÿÿÿÿÿB *ÿÿ ÿz)SoSObfuscationArchive.obfuscate_arc_filescCst‚dS)z=Check if the archive is a well-known type we directly supportN)ÚNotImplementedError)Úclsr?rrr Ú check_is_typeìsz#SoSObfuscationArchive.check_is_typecCsd|jj ¡vS)Nr!)Ú __class__Ú__name__Úlower©r3rrr Úis_sosñszSoSObfuscationArchive.is_soscCs d|jvS)NZinsights)Ú type_namerrrrr Ú is_insightsõsz!SoSObfuscationArchive.is_insightscCs|jrt |j¡|_dSr)Ú is_tarfilerrrÚtarobjrrrrr r,ùsz SoSObfuscationArchive._load_selfcCsgS)a/Return a list of ObfuscationArchives that represent additional archives found within the target archive. For example, an archive from `sos collect` will return a list of ``SoSReportArchive`` objects. This should be overridden by individual types of ObfuscationArchive's rrrrrr Úget_nested_archivesþsz)SoSObfuscationArchive.get_nested_archivescCs>|jr0|jj}| ¡r|jStj |j¡p.tjStj  |j ¡S)z|Set the root path for the archive that should be prepended to any filenames given to methods in this class. ) rvrwZ firstmemberÚisdirrrr ÚdirnameÚseprr)r3Ztoplevelrrr Úget_archive_roots z&SoSObfuscationArchive.get_archive_rootcCs"|j |jdd›d|›¡dS)z9Helper to easily format ui messages on a per-report basisz :z<50ú N)r(Úinfor%)r3Úmsgrrr Ú report_msgsz SoSObfuscationArchive.report_msgcCs&d|rd|›nd›d|j›d|›S)Nz[cleanerú:r"ú[z]] )r$©r3rZcallerrrr Ú _fmt_log_msgs ÿÿz"SoSObfuscationArchive._fmt_log_msgcCs|j | ||¡¡dSr)r'Údebugr„rƒrrr rDszSoSObfuscationArchive.log_debugcCs|j | ||¡¡dSr)r'r~r„rƒrrr r1szSoSObfuscationArchive.log_infocCs|j | ||¡¡dSr)r'Úerrorr„rƒrrr Ú log_error szSoSObfuscationArchive.log_errorcCsgd¢S)zyProvide a list of files and file regexes to skip obfuscation on Returns: list of files and file regexes )z proc/kallsymsz sosreport-z sys/firmwarezsys/fszsys/kernel/debugz sys/modulerrrrrr r)#sz%SoSObfuscationArchive._load_skip_listcCs(zt |j¡WSty"YdS0dS)NF)rrvrrrrrrr rv1s z SoSObfuscationArchive.is_tarfilecCs@|›d}| d|›d¡tdt|ƒ›|dt |¡|S)z›Convert a certificate to text. This is used when cleaner encounters a certificate file and the option 'treat_certificates' is 'obfuscate'. z.textzConverting certificate file 'z ' to textz%openssl storeutl -noout -text -certs )Zto_file)r1rÚstrrr<)r3rjZout_fnrrr r_8s  þ z)SoSObfuscationArchive.certificate_to_textcCs<| |¡}|r8| d|›d¡t |¡|jd7_dS)zˆRemove a file from the archive. This is used when cleaner encounters a binary file, which we cannot reliably obfuscate. zRemoving binary file 'z' from archiveéN)Ú get_file_pathr1rr<rh)r3rjZ full_fnamerrr r^Ds   z!SoSObfuscationArchive.remove_filecCs6|js&|js| ¡|_tj |j|¡Stj |j|¡S)zºBased on the type of archive we're dealing with, do whatever that archive requires to a provided **relative** filepath to be able to access it within the archive )r+r-r|rr rr[)r3rjrrr Úformat_file_nameOs  z&SoSObfuscationArchive.format_file_namec Csà|jdur\|jr\| |¡}z|j |¡ ¡ d¡WStyX| d|›d¡YdS0n€zBt | |¡ddd}| ¡WdƒWS1s’0YWn<t yÚ}z$| d |›d |›¡WYd}~dSd}~00dS) zÀReturn the content from the specified fname. Particularly useful for tarball-type archives so we can retrieve prep file contents prior to extracting the entire archive FrUzUnable to retrieve z: no such file in archiver"rT)rVNzFailed to get contents of rX) r+rvr‹rwZ extractfileÚreadÚdecodeÚKeyErrorrDrr)r3rjr>Zto_readr6rrr Úget_file_contentZs$   ÿ  ÿ,z&SoSObfuscationArchive.get_file_contentc Csj|jr,|s| d¡| ¡|_d|_d|_n|j|_t ¡dkrT|  d¡t  |j¡D]ú\}}}z¸|D]0}tj   ||¡}t  |¡j}t ||t jB¡qh|D]x}tj   ||¡} tj  | ¡ržtj  | ¡rÊqžt | tj¡ræt | tj¡sž|  d|  |j¡d›¡t | t jt jB¡qžWqXtyP} z|  d| ›¡WYd} ~ qXd} ~ 00qX|  d|j›¡dS) Nz Extracting...Trz)Verifying permissions of archive contentszAdding owner rw permissions to rz!Error while trying to set perms: zExtracted path is )rvr€Ú extract_selfr[r+rwrrÚgetuidrDÚwalkr rÚstatÚst_modeÚchmodÚS_IRWXUÚexistsr9ÚaccessÚR_OKÚW_OKrÚS_IRUSRÚS_IWUSRr) r3ÚquietrzÚdirsÚfilesÚ_dirZ_dirnameZ _dir_permsr>rjr6rrr rqs>     ÿÿÿ(zSoSObfuscationArchive.extractcCs.|j |j|¡}||_t |j|¡||_dS)zÐRename the top-level directory to new_name, which should be an obfuscated string that scrubs the hostname from the top-level dir which would be named after the unobfuscated sos report N)r[r8r$rr:)r3Únew_nameÚ_pathrrr Úrename_top_dir—sz$SoSObfuscationArchive.rename_top_dircCs|jr|j d¡rdSdSdS)z¸Return the compression type used by the archive, if any. This is then used by SoSCleaner to generate a policy-derived compression command to repack the archive ÚxzÚgzN)rvrÚendswithrrrrr Úget_compression¡s  z%SoSObfuscationArchive.get_compressioncCs´d}|jd}i}|rL|d|›7}|d|›7}|dkrDddi}ndd i}| d |›¡tj|fd |i|¤Ž.}|j|jtj |j¡d d Wdƒn1s¦0Y|S)zIPack the extracted archive as a tarfile to then be re-compressed rQz-obfuscated.tarrÚ.r¤ÚpresetéÚ compresslevelézBuilding tar file rRr‰)ZarcnameN) r[rDrrÚaddrr rr$)r3ÚmethodrRZtarpathZ compr_argsÚtarrrr Úbuild_tar_file¬s  ÿ$z$SoSObfuscationArchive.build_tar_filec Csªz| |¡|_Wn6tyF}z| d|›¡‚WYd}~n d}~00| d|j›¡z | ¡Wn>ty¤}z&| d|›¡| d¡WYd}~n d}~00dS)z•Execute the compression command, and set the appropriate final archive path for later reference by SoSCleaner on a per-archive basis z(Exception while re-compressing archive: NzCompressed to z'Failed to remove extraction directory: z/Failed to remove temporary extraction directory)r°r#rrDÚremove_extracted_pathr€)r3r®r6rrr Úcompress¿s zSoSObfuscationArchive.compresscCsrz"| d|j›¡t |j¡WnJtylt |jtj¡tj   |j¡r\t  |j¡n t |j¡Yn0dS)z¦After the tarball has been re-compressed, remove the extracted path so that we don't take up that duplicate space any longer during execution z Removing N) rDr[rcÚrmtreeÚOSErrorrr•r“rœr Úisfiler<rrrrr r±Ïs z+SoSObfuscationArchive.remove_extracted_pathcCsJtdƒ.}| t|j|j¡}| ¡}|WdƒS1s<0YdS)z}Extract an archive into our tmpdir so that we may inspect it or iterate through its contents for obfuscation r‰N)rZsubmitrrrÚresult)r3Z_poolZ _path_futurer rrr rÞs  ÿz"SoSObfuscationArchive.extract_selfccspt |j¡D]^\}}}|D]$}tj ||¡}tj |¡r|Vq|D]$}tj ||¡}tj |¡rD|VqDq dS)z.Iterator for a list of symlinks in the archiveN)rr’r[r rr9)r3rzržrŸr Z_dirpathr>Ú_fnamerrr Ú get_symlinksés  z"SoSObfuscationArchive.get_symlinksccsLt |j¡D]:\}}}|D]*}tj || d¡¡}tj |¡s|Vqq dS)z¡Iterator for a list of files in the archive, to allow clean to iterate over. Will not include symlinks, as those are handled separately r N)rr’r[r rÚlstripr9)r3rzÚ_rŸr>r·rrr Ú get_filesõs  zSoSObfuscationArchive.get_filescCs*g}t |j¡D]\}}}| |¡q|S)z3Return a list of all directories within the archive)rr’r[Úappend)r3Zdir_listrzrºrrr Úget_directory_lists z(SoSObfuscationArchive.get_directory_listcCs |jd7_|j|7_dS)z’Called when a file has finished being parsed and used to track total substitutions made and number of files that had changes made r‰N)rfrg)r3rFrrr resz&SoSObfuscationArchive.update_sub_countcCs*tj |j| d¡¡}tj |¡r&|SdS)zReturn the filepath of a specific file within the archive so that it may be selectively inspected if it exists r r")rr rr[r¹r—)r3rjr¢rrr rŠsz#SoSObfuscationArchive.get_file_pathcCsTtj | |¡¡s(tj | |¡¡s(dS|jD] }| |¡sHt ||¡r.dSq.dS)a%Checks the provided filename against a list of filepaths to not perform obfuscation on, as defined in self.skip_list Positional arguments: :param filename str: Filename relative to the extracted archive root TF) rr rµrŠr9r*Ú startswithÚrerJ)r3r>rLrrr r\s ÿ z&SoSObfuscationArchive.should_skip_filecCsFgd¢}|D]}t ||¡r dSq | |¡}tj |¡rBt|ƒSdS)aaDetermine if the file should be removed or not, due to an inability to reliably obfuscate that file based on the filename. :param fname: Filename relative to the extracted archive root :type fname: ``str`` :returns: ``True`` if the file cannot be reliably obfuscated :rtype: ``bool`` ) z.*\.gz$z.*\.xz$z .*\.bzip2$z .*\.tar\..*z.*\.txz$z.*\.tgz$z.*\.bin$z .*\.journal$z.*\~$TF)r¿rJrŠrr rµr)r3rjZobvious_removesZ_arc_regZ _full_pathrrr r])s    z(SoSObfuscationArchive.should_remove_file)N)N)N)N)N)F)3rpÚ __module__Ú __qualname__Ú__doc__rfrgrhrtr2Z is_nestedZ prep_filesr4r7r@rArBrHrkÚ classmethodrnÚpropertyrsrur,rxr|r€r„rDr1r‡r)rvr_r^r‹rrr£r§r°r²r±rr¸r»r½rerŠr\r]rrrr r 3sd  Q              &     r )r&rrcr“rr`r¿Úconcurrent.futuresrZ sos.utilitiesrrrrr rrrr Ú s