4

I run a website where aspiring writers post their writing. Naturally, some of this writing can be of a 'sensitive' nature. Sometimes it contains a lot of swear words, sometimes it's malicious, sometimes it's suicidal. It's not the norm, but it exists.

Of course, we want our users to feel free to post what content they like, but the issue arises with advertisers. Namely, Google Adsense, which routinely sends us warnings about a piece of content that contains too much swearing or is defamatory. The latest is a piece someone wrote about cutting themselves which Google has deemed too 'tragic'.

I've generally been able to write scripts too look through the text and detect a lot of swear words or racist terms or anti-gay rhetoric, etc. But I can't for the live of me figure out how to detect 'tragic' text.

So, the question is 2 fold.

  1. Is there some sort of PHP class/function/API that is good at detecting inappropriate content?
  2. Any idea on how to automatically detect suicidal or self-harming content?
5
  • 2
    Nothing beats human moderation, I guess. I don't think even Google can do this completely automatically, either. Maybe something that detects common words or patterns, and then submits the content to a human reviewer
    – Pekka
    Commented Mar 22, 2013 at 17:05
  • Enforce a category system onto the site? example: Dark, Fantasy, etc.. -- Then you can search through the categories and edit as seen fit. The only other way I can suggest, is create a function using str_pos
    – Daryl Gill
    Commented Mar 22, 2013 at 17:11
  • 2
    Automating this kind of thing is widely recognised to be next to impossible (Google "clbuttic"). Natural languages are too varied to adequately detect every possible pattern, and with the use case you describe it is likely that some items you would filter would be using words that are less common, and possibly even other languages you do not look for. The only option that you have (the way I see it) is to introduce a human-powered moderation system. Possibly one thing you could do would be to publish immediately but not show any advertising with a given article until it has been reviewed.
    – DaveRandom
    Commented Mar 22, 2013 at 17:18
  • 3
    It's too much; I can't deal with it any more. I have to finally put and end to it, gather all my strength and do it. I can't live with myself the way my lawn looks right now, I have to cut it. (not to make light of suicidal tendencies, but you see how hard it will be to detect meaning automatically)
    – Pekka
    Commented Mar 22, 2013 at 17:39
  • Thanks for the suggestions. Human moderation and user flagging is probably the way to go. For now, though, I've just built a simple script to check the words in the text for suicidal flags (slit wrists, self harm, end it all, please forgive me, etc.). If a piece of writing meets a certain threshold where it has more than a certain number of those phrases or the percent is over a certain point I don't show Google Ads. P.S. Making a list of suicidal phrases was very depressing.
    – Charlie
    Commented Mar 22, 2013 at 20:18

1 Answer 1

2

You could train a Bayesian filter on what tragic content looks like. Like a spam filter, but for the content you want to keep out. I've used this PHP library and it works well: https://github.com/Dachande663/PHP-Classifier

It would fit in well with the human moderation and flagging process.

1
  • Hi @popthestack, do you have any source which uses Bayesian filters for the task. thanks Commented Jun 15, 2018 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.