If you lowered your difficulty, other nodes who follow the prescribed convention for setting the difficulty will reject your blocks and any further chain built on them as invalid. Agreement on the formula for computing difficulty is required to have a unified block chain. If a bunch of nodes pick a different difficulty formula, it will fork the block chain, and there will be two different incompatible transaction records.