a Ö>ïh”Xã@sjddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z dd„Z Gdd„dƒZ dS)éN)ÚProcessPoolExecutor)Úfile_is_binaryc Csàt |¡Â}tj |d¡}ttddd„ƒ|_| ¡D]d}tj ||j¡}tj  |¡}tj  |¡}tj  ||g¡}||krŒt d|›d|›ƒ‚|  ||¡q4tj ||j  d¡d  d ¡d ¡WdƒS1sÒ0YdS) NZcleanerZfully_trusted_filtercSs|S©N©)ÚmemberÚpathrrúA/usr/lib/python3.9/site-packages/sos/cleaner/archives/__init__.pyÚ"óz!extract_archive..z"Attempted path traversal in tarflez != ú/éÿÿÿÿú.tarr)ÚtarfileÚopenÚosrÚjoinÚgetattrZextraction_filterZ getmembersÚnameÚabspathÚ commonprefixÚ ExceptionÚextractÚsplit) Ú archive_pathÚtmpdirÚarchiverrZ member_pathZ abs_directoryZ abs_targetÚprefixrrrÚextract_archives$ ÿ   ÿÿrc@speZdZdZdZdZdZdZdZdZ iZ dd„Z dd„Z d d „Z d d „Zd d„ZdPdd„Zdd„Zedd„ƒZedd„ƒZedd„ƒZdd„Zdd„Zdd„Zd d!„ZdQd"d#„ZdRd$d%„ZdSd&d'„ZdTd(d)„Zd*d+„Zed,d-„ƒZ d.d/„Z!d0d1„Z"d2d3„Z#dUd4d5„Z$d6d7„Z%d8d9„Z&d:d;„Z'dd?„Z)d@dA„Z*dBdC„Z+dDdE„Z,dFdG„Z-dHdI„Z.dJdK„Z/dLdM„Z0dNdO„Z1dS)VÚSoSObfuscationArchiveaA representation of an extracted archive or an sos archive build directory which is used by SoSCleaner. Each archive that needs to be obfuscated is loaded into an instance of this class. All report-level operations should be contained within this class. rZ undeterminedFcCs˜||_|j|_||_|j d¡d d¡d|_|j|_t d¡|_t d¡|_ |  ¡|_ d|_ |  ¡d|_||_d |_| d |j›d |j›¡dS) Nr r r rÚsosZsos_uiFÚrzLoaded z as type )rÚfinal_archive_pathrrÚ archive_nameÚui_nameÚloggingZ getLoggerÚsoslogÚui_logÚ_load_skip_listÚ skip_listÚ is_extractedÚ _load_selfÚ archive_rootÚkeep_binary_filesÚparsersÚlog_infoÚ description)Úselfrrr,rrrÚ__init__Bs    ÿzSoSObfuscationArchive.__init__c CsT|jD]H}z| |¡}WqtyL}z| d|›¡WYd}~qd}~00q|S)NzError obfuscating string data: )r-Zparse_string_for_keysrr.)r0Z string_dataÚparserÚerrrrrÚobfuscate_stringTs  (z&SoSObfuscationArchive.obfuscate_stringcCs’| | d¡d¡}| | d¡d|¡}||krŽ| |¡d}tj ||¡}tj |¡sht ||¡n&| t |¡¡}t  |¡t  ||¡dS)Nr r r) r4rÚreplacerrrÚislinkÚrenameÚreadlinkÚremoveÚsymlink)r0Ú short_nameÚfilenameZ_ob_short_nameZ _ob_filenameÚarc_pathZ_ob_pathZ _target_obrrrÚobfuscate_filename^sÿ  z(SoSObfuscationArchive.obfuscate_filenamecCs ||_dSr)r-)r0r-rrrÚ set_parsersusz!SoSObfuscationArchive.set_parserscCs|jD] }| ¡qdSr)r-Zload_map_entries)r0r2rrrÚload_parser_entriesxs z)SoSObfuscationArchive.load_parser_entriesNc Csˆd}| ¡s||fS|dur"|j}|D]X}z| |¡\}}||7}Wq&ty|}z | d|›|j¡WYd}~q&d}~00q&||fS)aÿRun a line through each of the obfuscation parsers, keeping a cumulative total of substitutions done on that particular line. Positional arguments: :param line str: The raw line as read from the file being processed :param parsers: A list of parser objects to obfuscate with. If None, use all. Returns the fully obfuscated line and the number of substitutions made rNzfailed to parse line: )Ústripr-Z parse_linerÚ log_debugr)r0Úliner-Úcountr2Ú_countr3rrrÚobfuscate_line|s ,z$SoSObfuscationArchive.obfuscate_linec sB|D]*}| dt ¡›d|›¡zÂ| |jd¡d‰| ˆ¡rJWq|jsh| ˆ¡rh| ˆ¡Wq|jr„t |ƒs€| ˆ¡r„Wqtj   |¡r”Wq‡fdd„|j Dƒ}|sÆ| dˆp¸|›d¡Wq| d ˆpÒ|›¡d }t jd |jd Ú}t|d ddd€}|D]j}z&| ||¡\}}||7}| |¡Wn<tyl} z"| dˆ›d| ›¡WYd} ~ n d} ~ 00qWdƒn1sˆ0Y| d ¡|rºt |j|¡| |¡Wdƒn1sÐ0Y| ˆ|¡Wqty.} z,| dt ¡›d|›d| ›¡WYd} ~ qd} ~ 00q|j|j|jfS)Nz pid=z: obfuscating r écs(g|] }t‡fdd„|jDƒƒs|‘qS)c3s|]}| ˆ¡VqdSr)Úmatch)Ú.0Ú_skip©r;rrÚ ²szGSoSObfuscationArchive.obfuscate_arc_files...)ÚanyZ skip_patterns)rIZ_prKrrÚ °s  ÿÿz=SoSObfuscationArchive.obfuscate_arc_files..zSkipping obfuscation of z" due to matching file skip patternz Obfuscating rÚw)ÚmodeÚdirÚrúutf-8r5)ÚencodingÚerrorszUnable to obfuscate ú: z': caught exception on obfuscating file )rBrÚgetpidrr"Úshould_skip_filer,Úshould_remove_fileÚ remove_filerrr6r-ÚtempfileZNamedTemporaryFilerrrFÚwriterÚseekÚshutilÚcopyfilerÚupdate_sub_countr>Úfiles_obfuscated_countÚtotal_sub_countÚremoved_file_count) r0Úflistr<Z_parsersZsubsZtfileÚfnamerCZcntr3rrKrÚobfuscate_arc_files˜sx  ÿ ÿþ  ÿÿÿÿÿB *ÿÿ ÿz)SoSObfuscationArchive.obfuscate_arc_filescCst‚dS)z=Check if the archive is a well-known type we directly supportN)ÚNotImplementedError)Úclsr=rrrÚ check_is_typeØsz#SoSObfuscationArchive.check_is_typecCsd|jj ¡vS)Nr)Ú __class__Ú__name__Úlower©r0rrrÚis_sosÝszSoSObfuscationArchive.is_soscCs d|jvS)NZinsights)Ú type_namermrrrÚ is_insightsász!SoSObfuscationArchive.is_insightscCs|jrt |j¡|_dSr)Ú is_tarfilerrrÚtarobjrmrrrr*åsz SoSObfuscationArchive._load_selfcCsgS)a/Return a list of ObfuscationArchives that represent additional archives found within the target archive. For example, an archive from `sos collect` will return a list of ``SoSReportArchive`` objects. This should be overridden by individual types of ObfuscationArchive's rrmrrrÚget_nested_archivesêsz)SoSObfuscationArchive.get_nested_archivescCs>|jr0|jj}| ¡r|jStj |j¡p.tjStj  |j ¡S)z|Set the root path for the archive that should be prepended to any filenames given to methods in this class. ) rqrrZ firstmemberÚisdirrrrÚdirnameÚseprr)r0ZtoplevelrrrÚget_archive_rootós z&SoSObfuscationArchive.get_archive_rootcCs"|j |jdd›d|›¡dS)z9Helper to easily format ui messages on a per-report basisz :z<50ú N)r&Úinfor#)r0ÚmsgrrrÚ report_msgþsz SoSObfuscationArchive.report_msgcCs&d|rd|›nd›d|j›d|›S)Nz[cleanerú:r ú[z]] )r"©r0rzZcallerrrrÚ _fmt_log_msgs ÿÿz"SoSObfuscationArchive._fmt_log_msgcCs|j | ||¡¡dSr)r%Údebugrr~rrrrBszSoSObfuscationArchive.log_debugcCs|j | ||¡¡dSr)r%ryrr~rrrr. szSoSObfuscationArchive.log_infocCs|j | ||¡¡dSr)r%Úerrorrr~rrrÚ log_error szSoSObfuscationArchive.log_errorcCsgd¢S)zyProvide a list of files and file regexes to skip obfuscation on Returns: list of files and file regexes )z proc/kallsymsz sosreport-z sys/firmwarezsys/fszsys/kernel/debugz sys/modulerrmrrrr'sz%SoSObfuscationArchive._load_skip_listcCs(zt |j¡WSty"YdS0dS)NF)rrqrrrmrrrrqs z SoSObfuscationArchive.is_tarfilecCs<| |¡}|r8| d|›d¡t |¡|jd7_dS)zˆRemove a file from the archive. This is used when cleaner encounters a binary file, which we cannot reliably obfuscate. zRemoving binary file 'z' from archiverGN)Ú get_file_pathr.rr9rc)r0reZ full_fnamerrrrZ$s   z!SoSObfuscationArchive.remove_filecCs6|js&|js| ¡|_tj |j|¡Stj |j|¡S)zºBased on the type of archive we're dealing with, do whatever that archive requires to a provided **relative** filepath to be able to access it within the archive )r)r+rwrrrÚextracted_path)r0rerrrÚformat_file_name/s  z&SoSObfuscationArchive.format_file_namec Csà|jdur\|jr\| |¡}z|j |¡ ¡ d¡WStyX| d|›d¡YdS0n€zBt | |¡ddd}| ¡WdƒWS1s’0YWn<t yÚ}z$| d |›d |›¡WYd}~dSd}~00dS) zÀReturn the content from the specified fname. Particularly useful for tarball-type archives so we can retrieve prep file contents prior to extracting the entire archive FrSzUnable to retrieve z: no such file in archiver rR)rTNzFailed to get contents of rV) r)rqr…rrZ extractfileÚreadÚdecodeÚKeyErrorrBrr)r0rer<Zto_readr3rrrÚget_file_content:s$   ÿ  ÿ,z&SoSObfuscationArchive.get_file_contentc Csj|jr,|s| d¡| ¡|_d|_d|_n|j|_t ¡dkrT|  d¡t  |j¡D]ú\}}}z¸|D]0}tj   ||¡}t  |¡j}t ||t jB¡qh|D]x}tj   ||¡} tj  | ¡ržtj  | ¡rÊqžt | tj¡ræt | tj¡sž|  d|  |j¡d›¡t | t jt jB¡qžWqXtyP} z|  d| ›¡WYd} ~ qXd} ~ 00qX|  d|j›¡dS) Nz Extracting...Trz)Verifying permissions of archive contentszAdding owner rw permissions to r z!Error while trying to set perms: zExtracted path is )rqr{Ú extract_selfr„r)rrrrÚgetuidrBÚwalkrrÚstatÚst_modeÚchmodÚS_IRWXUÚexistsr6ÚaccessÚR_OKÚW_OKrÚS_IRUSRÚS_IWUSRr) r0ÚquietruÚdirsÚfilesÚ_dirZ_dirnameZ _dir_permsr<rer3rrrrQs>     ÿÿÿ(zSoSObfuscationArchive.extractcCs.|j |j|¡}||_t |j|¡||_dS)zÐRename the top-level directory to new_name, which should be an obfuscated string that scrubs the hostname from the top-level dir which would be named after the unobfuscated sos report N)r„r5r"rr7)r0Únew_nameÚ_pathrrrÚrename_top_dirwsz$SoSObfuscationArchive.rename_top_dircCs|jr|j d¡rdSdSdS)z¸Return the compression type used by the archive, if any. This is then used by SoSCleaner to generate a policy-derived compression command to repack the archive ÚxzÚgzN)rqrÚendswithrmrrrÚget_compressions  z%SoSObfuscationArchive.get_compressioncCs´d}|jd}i}|rL|d|›7}|d|›7}|dkrDddi}ndd i}| d |›¡tj|fd |i|¤Ž.}|j|jtj |j¡d d Wdƒn1s¦0Y|S)zIPack the extracted archive as a tarfile to then be re-compressed rOz-obfuscated.tarr|Ú.ržÚpresetéÚ compresslevelézBuilding tar file rPrG)ZarcnameN) r„rBrrÚaddrrrr")r0ÚmethodrPZtarpathZ compr_argsÚtarrrrÚbuild_tar_fileŒs  ÿ$z$SoSObfuscationArchive.build_tar_filec Csªz| |¡|_Wn6tyF}z| d|›¡‚WYd}~n d}~00| d|j›¡z | ¡Wn>ty¤}z&| d|›¡| d¡WYd}~n d}~00dS)z•Execute the compression command, and set the appropriate final archive path for later reference by SoSCleaner on a per-archive basis z(Exception while re-compressing archive: NzCompressed to z'Failed to remove extraction directory: z/Failed to remove temporary extraction directory)rªr!rrBÚremove_extracted_pathr{)r0r¨r3rrrÚcompressŸs zSoSObfuscationArchive.compresscCsrz"| d|j›¡t |j¡WnJtylt |jtj¡tj   |j¡r\t  |j¡n t |j¡Yn0dS)z¦After the tarball has been re-compressed, remove the extracted path so that we don't take up that duplicate space any longer during execution z Removing N) rBr„r^ÚrmtreeÚOSErrorrrrr–rÚisfiler9rmrrrr«¯s z+SoSObfuscationArchive.remove_extracted_pathcCsJtdƒ.}| t|j|j¡}| ¡}|WdƒS1s<0YdS)z}Extract an archive into our tmpdir so that we may inspect it or iterate through its contents for obfuscation rGN)rZsubmitrrrÚresult)r0Z_poolZ _path_futurerrrrrŠ¾s  ÿz"SoSObfuscationArchive.extract_selfccspt |j¡D]^\}}}|D]$}tj ||¡}tj |¡r|Vq|D]$}tj ||¡}tj |¡rD|VqDq dS)z.Iterator for a list of symlinks in the archiveN)rrŒr„rrr6)r0rur˜r™ršZ_dirpathr<Ú_fnamerrrÚ get_symlinksÉs  z"SoSObfuscationArchive.get_symlinksccsLt |j¡D]:\}}}|D]*}tj || d¡¡}tj |¡s|Vqq dS)z¡Iterator for a list of files in the archive, to allow clean to iterate over. Will not include symlinks, as those are handled separately r N)rrŒr„rrÚlstripr6)r0ruÚ_r™r<r±rrrÚ get_filesÕs  zSoSObfuscationArchive.get_filescCs*g}t |j¡D]\}}}| |¡q|S)z3Return a list of all directories within the archive)rrŒr„Úappend)r0Zdir_listrur´rrrÚget_directory_listás z(SoSObfuscationArchive.get_directory_listcCs |jd7_|j|7_dS)z’Called when a file has finished being parsed and used to track total substitutions made and number of files that had changes made rGN)rarb)r0rDrrrr`èsz&SoSObfuscationArchive.update_sub_countcCs*tj |j| d¡¡}tj |¡r&|SdS)zReturn the filepath of a specific file within the archive so that it may be selectively inspected if it exists r r )rrrr„r³r‘)r0rerœrrrrƒïsz#SoSObfuscationArchive.get_file_pathcCsTtj | |¡¡s(tj | |¡¡s(dS|jD] }| |¡sHt ||¡r.dSq.dS)a%Checks the provided filename against a list of filepaths to not perform obfuscation on, as defined in self.skip_list Positional arguments: :param filename str: Filename relative to the extracted archive root TF) rrr¯rƒr6r(Ú startswithÚrerH)r0r<rJrrrrXös ÿ z&SoSObfuscationArchive.should_skip_filecCsFgd¢}|D]}t ||¡r dSq | |¡}tj |¡rBt|ƒSdS)aaDetermine if the file should be removed or not, due to an inability to reliably obfuscate that file based on the filename. :param fname: Filename relative to the extracted archive root :type fname: ``str`` :returns: ``True`` if the file cannot be reliably obfuscated :rtype: ``bool`` ) z.*\.gz$z.*\.xz$z .*\.bzip2$z .*\.tar\..*z.*\.txz$z.*\.tgz$z.*\.bin$z .*\.journal$z.*\~$TF)r¹rHrƒrrr¯r)r0reZobvious_removesZ_arc_regZ _full_pathrrrrY s    z(SoSObfuscationArchive.should_remove_file)N)N)N)N)N)F)2rkÚ __module__Ú __qualname__Ú__doc__rarbrcror/Z is_nestedZ prep_filesr1r4r>r?r@rFrfÚ classmethodriÚpropertyrnrpr*rsrwr{rrBr.r‚r'rqrZr…r‰rrr¡rªr¬r«rŠr²rµr·r`rƒrXrYrrrrr2sb  @             &     r) r$rr^rrr[r¹Úconcurrent.futuresrZ sos.utilitiesrrrrrrrÚ s