Tag and clean data files

The job is to add tags the the data sourced from [login to view URL] Raw data is attached as: [login to view URL] Our requirements are as follows: 1. only the stories (i.e. NO metadata e.g. author/book details, preface, postscript, contents, chapter headings, foot notes) 2. superfluous text removed (i.e. any non story text e.g. quotes, illustration markers, page markers -> e.g. * * * * *)) Tagging requirements: - Point 1 (as above) - add <|startoftext|> tag at the beginning and <|endoftext|> tag at the story text - Point 2 (as above) - delete any superfluous text between the added <|startoftext|> and <|endoftext|> tags Note: Each file may contain one or more stories. Examples (attached): Pre-processing - A Book of Myths by Jean Lang - Free Ebook - [login to view URL] Post-processing - A Book of Myths by Jean Lang - Free Ebook - [login to view URL] Note: Additional requirements: - Please save the original and tagged files with the postfix as illustrated in the attached examples. - Verify correct tagging after completing the first file and before continuing with the remaining files. In our experience to date with this work, this job should take 7-8 hours. Successful completion of this work in the expected timeframe will lead to more of the same (total approx 70-80 hrs work).