Mysql deleted logs consuming physical server space

If you find that the server is using excess amount of physical disk space;

try a df -h and find out the partition where the disk usage is getting higher.

root@server:~# df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/vda1       242G  194G   36G  85% /

udev             10M     0   10M   0% /dev

tmpfs           6.3G   41M  6.3G   1% /run

tmpfs            16G     0   16G   0% /dev/shm

tmpfs           5.0M     0  5.0M   0% /run/lock

tmpfs            16G     0   16G   0% /sys/fs/cgroup

root@server:~#

Gere we can see that the / partition is using much space than it should be.

confirm the situation by changing intho the / partition and do a 'du'

root@server:# du -sch /*

11M     /bin

30M     /boot

0       /dev

5.5M    /etc

24K     /home

24K     /tmp

1.2G    /usr

23G     /var

0       /vmlinuz

25G     total

Here we can see that the total space used by / partition is just 25 GB and not 194GB as per the result of 'df' command.

Than means some deleted files are beling held by some application .

To verify that run the following.

root@server# lsof | grep deleted | numfmt --field=8 --to=iec | head -n 50

In the result you will find a line as 

mysqld     9614  9616       mysql   10w      REG              254,1         170G    2622664 /var/log/mysql/mysql.log.1 (deleted)

here we can see that the mysql process is holding up the missing 170GB log file which is already deleted.

Run the following command to verify  in which the 9616 is the process ID and 10 is the fd numer , both taken from the result of the above command.

root@server# file /proc/9616/fd/10
/proc/9616/fd/10: broken symbolic link to /var/log/mysql/mysql.log.1 (deleted)

Do an echo to the file that it can be cleared from space utilized.

root@server# echo > /proc/9616/fd/10

Afterwards you wont find it when rerun lsof command.
And when you check the disk usage using the df-h command , you can see that the mising space is returned.