Like others have said. It's probably between your house and the street main.
If the clean out is just outside the kitchen wall, make it backup again and remove the cap. I'd assume sewer water will flow out.
I'd try and rent a bigger snake and run it from the clean out to the road. Any big...